fix: optimize segment compaction by reusing first vector index#114
fix: optimize segment compaction by reusing first vector index#114kgeg401 wants to merge 3 commits intoalibaba:mainfrom
Conversation
|
Thank you for your contribution. The zvec maintainer team is currently on holiday for the Chinese New Year (Spring Festival) in mainland China. Please expect delayed responses, and we appreciate your patience. |
0cb23df to
98d5708
Compare
98d5708 to
17581fd
Compare
|
Added a quantized-path regression test for segment compaction:\n\n- New CompactTask_QuantizedVectorIndexThreeSegmentsRegression in segment_helper_test.cc.\n- Covers multi-segment, no-filter compaction with quantized vector indexes and verifies quantized index availability after compaction.\n\nThis is intended to directly exercise the get_quant_vector_indexer(...) merge-reuse path. |
|
Added the quantized-path regression test for segment compaction, and the PR is otherwise unchanged. This should be ready for maintainer review when bandwidth allows. |
|
Thanks a lot for the contribution, and sorry for the long delay on this review — we had some internal bandwidth constraints. We've made PR/issue handling to a regular cadence :) On the change itself: I feel this optimization is better placed in the core reducer layer( Also, nice improvement on deduplicating repeated merge logic. One more cleanup suggestion: merge_options construction could likely be moved out of the lambda and built once outside the loop. |
Summary
Root Cause
ReduceVectorIndex always rebuilt merged indexes from scratch, even when compaction had no filter and segments were layout-compatible. That caused avoidable index rebuild work during merge/compact.
Validation
Fixes #98